Fix 500s and empty results from binary fields in Calcite pushdowns - #5767

Description

source=idx | sort bin on a binary field returns a 500 whose reason is a generic Failed to fetch data from the index. The real cause is visible only in details, where it reads IllegalArgumentException[Can't load fielddata on [bin] because fielddata is unsupported on fields of type [binary]]. Nine commands fail this way. A tenth, where isnotnull(bin), returns 200 with zero rows against documents that all have the field populated.

A binary field has neither fielddata nor doc values, so OpenSearch cannot bucket or sort on it. The plan-time guards ask whether a type is atomic, not whether it is aggregatable. Binary("binary", ExprCoreType.UNKNOWN) reads as atomic, passes every guard, and reaches the shard. geo_point becomes GEOMETRY, fails the same test, and is already rejected as a 400 by #5751. binary is not.

The filter case fails differently, and silently. existsQuery(bin) is valid DSL, but BinaryFieldMapper indexes nothing when doc_values is false, not even a _field_names entry, so exists has no term to match and the shard honestly reports zero hits for a correctly-formed query.

This refuses a binary reference where a field reference resolves, so each pushdown declines and Calcite keeps an un-pushed plan that returns correct results.

• PredicateAnalyzer.NamedFieldExpression.getReference() and getReferenceForTermQuery(), the seam every affected family already routes through. The filter path re-analyzes the predicate as a _source script and stays pushed down, while the aggregate and sort paths decline the planner rule.
• AbstractCalciteIndexScan, the two field-sort sites. The script-sort branch beside them needs no guard, since it reads from _source.
• AggregateAnalyzer, the dedup sort hint, which carries a raw field name that never reaches the accessors above.
• RexStandardizer, route a binary field to _source rather than doc values, which is what makes the filter redirect work.

Refusing in the accessors rather than in the NamedFieldExpression constructors is deliberate. A top_hits fetch field only needs getRootName(), and BinaryFieldMapper.BinaryFieldType.valueFetcher returns SourceValueFetcher.identity, so the fields API serves a binary field from _source and that request shape was always valid. A constructor-level refusal would decline dedup on any index whose mapping merely contains a binary field, even when the query never names it, costing a pushdown that works correctly today, 0.0075s pushed down against 0.254s declined over 50000 documents. The last case in the test file pins this.

Before

"reason":  "Failed to fetch data from the index: the background task failed or interrupted."
"details": "... IllegalArgumentException[Can't load fielddata on [bin] because fielddata is
            unsupported on fields of type [binary]. Use doc values instead.]"
"status":  500

After, the same query, HTTP 200

"schema":   [{"name": "host", "type": "string"}, {"name": "bin", "type": "binary"}]
"datarows": [["host-a", "Y210"], ["host-a", "Y211"], ["host-b", "Y212"], ["host-b", "Y213"]]
"total":    4

Behaviour on a binary field, measured on a live cluster before and after.

One behaviour change beyond the reported symptom. The RexStandardizer change applies to every script context, so a pushed-down script referencing a binary field previously read null from doc values and now reads the real base64 value. eval x = concat(bin, 'a') returns a value where it used to return null.

That also moves where a bad script fails. A comparison against a binary field is now compiled on the shard against the real value, so where bin = 'zzz' returns a QueryShardException for a script it cannot compile rather than the earlier fielddata error. Nothing regresses, but the error text changes.

graphLookup on a binary edge field is the one caller these accessors do not protect, and it was already broken. CalciteEnumerableGraphLookup.queryLookupTable resolves the edge field at execution time outside any decline path, so the refusal escapes as a 500 instead of declining a rule. It returned a 500 before this change as well, since the terms query it emitted on a binary field fails at the shard, so the only difference is that the message now names the field. Verified on a live cluster, a keyword edge returns rows and a binary edge returns 500 either side of the change. graphLookup is marked experimental and a base64 blob is not a plausible graph edge, so this is recorded, not fixed.

An alias field whose path is a binary field looks unguarded and is not. instanceof OpenSearchBinaryType is false for the OpenSearchAliasType such a mapping produces, but Calcite resolves the alias at plan time to the base field's input ref, so the explain shows payload_alias=[$1] where $1 is payload, and the name reaching the guard is never the alias. Verified on a live cluster.

Out of scope, noted in the issue. Comparing a binary field against a string still fails, and not only for =. where bin = 'zzz' and where bin != 'zzz' report Cannot cast "java.lang.String" to "org.apache.calcite.avatica.util.ByteString", where bin > 'a' reports no applicable SqlFunctions.gt overload, and where match(bin, 'zzz') cannot work at all against a field OpenSearch does not index. Every one of those fails with pushdown disabled too, so declining the pushdown exposes the coercion gap rather than causing it, and the family belongs with #5753. What this fixes on the filter side is the null predicates, isnotnull(bin) returning all populated rows instead of none and isnull(bin) returning none, which is what the test file covers.

Also, stats count() by bin declines rather than taking the scripted _source route AggregateAnalyzer already uses for a text field with no .keyword. That costs a full scan, measured at 0.42s against 0.17s for a keyword group key over the same 50000 documents, so roughly 2.5x on a different cardinality rather than a like-for-like comparison. Worth it against a 500, but worth reclaiming. A blanket null return is not the way, because it would make max(bin) build a top_hits with no sort and return an arbitrary document, so keeping that pushdown needs the value-source site separated from the sort sites.

Related Issues

Resolves #5757

Testing

• integ-test/src/yamlRestTest/resources/rest-api-spec/test/issues/5757.yml, 19 cases over HTTP against a real binary mapping, covering all three pushdown families plus the xyseries aggregate-filter-argument route and the dedup sort hint. 18 assert answers. The last asserts a plan, because the accessor-versus-constructor placement above returns identical correct rows either way and is invisible to an answer assertion.
• Both new guards were confirmed to fail with their fix reverted and to pass with it restored. Removing the AggregateAnalyzer dedup-hint check fails only the dedup sort-hint case. Moving the refusal into the constructors fails only the plan case.
• RelJsonSerializerTest.testSerializeAndDeserializeUDT changes one expected value, the script source for the binary field, from DOC_VALUE to SOURCE. That flip is the unit-level assertion of the RexStandardizer change and of the behaviour change noted above, so it is the point of the edit rather than a fixup around it.

Verified against a local 3.9.0-SNAPSHOT tarball, single node and single shard, with plugins.calcite.enabled and plugins.calcite.pushdown.enabled both true. Not tested multi-shard or with security enabled.

Check List

• New functionality includes testing.
• New functionality has been documented.
• New functionality has javadoc added.
• New functionality has a user manual doc added.
• New PPL command checklist all confirmed.
• API changes companion pull request created.
• Commits are signed per the DCO using --signoff or -s.
• Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.